Learning to Reconstruct 3D Human Pose and Motion from Silhouettes

We will describe our ongoing work on learning-based methods for recovering 3D human body pose and motion from single images and from monocular image sequences. The methods work directly with raw image observations and require neither an explicit 3D body model nor a prior labelling of body parts in the image. Instead, they recover the body pose or motion by direct nonlinear regression against shape descriptors extracted automatically from image silhouettes or contours. For improved resistance to segmentation errors and occlusions, we use a robust shape representation: histograms of locally-supported shape-contexts descriptors. The image description is thus related to (generalized / robustified versions of) Brand’s ‘Shadow Puppetry’ [1], Mori & Malik’s shape context method [2] and Shakhnarovich et al’s contour based method [3]. We regress the current pose (body joint angles) against both silhouette shape and (in the tracking-based schemes) the previous 1–2 poses. The regression is nontrivial owing to high dimensionality, sparse training data, and the fact that recovering pose from monocular image observations is inherently multi-valued owing to pervasive kinematic ambiguities. For tracking, we evaluated several different regression dependency structures designed to reduce these reconstruction ambiguities while capturing the dynamics, observations and the correcting effect of observation updates (all nonlinear and unknown a priori). We tested a number of different regression methods on these problems, including regularized least squares, Support Vector Regression and Relevance Vector Machine (RVM) regression [4], over both linear and kernel bases. In general the kernelized methods do best and the RVM framework provides much sparser regressors without compromising performance. But linear least squares (over our very nonlinear shape description) also performs surprisingly well. If time permits, we will also sketch our novel scalable continuation-based RVM training algorithm. The methods are trained using real human motion capture data, to ensure that they capture both the global structure and the fine details of human motion. However to improve model coverage and make the most of the limited amount of training data available, we currently re-synthesize the corresponding training images from a range of different viewpoints. We have tested our models both quantitatively on independently captured test sequences and qualitatively on videos of typical human motions. On the test sequences, we are currently getting mean angular errors of about 6–7 degrees — a factor of about 3 better than the current state of the art for the much simpler upper-body-only problem.